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INTRODUCTION 

A core component of many new educational programs funded under Bush 
administration's No Child Be Left Behind Act of 2001 is that they be based on 
scientifically based research. Further emphasizing the important role that this type of 
research will play in future U.S. Department of Education priorities has been the 
establishment of the What Works Clearinghouse. Under the leadership of Dr. Grover 
Whitehurst, this new clearinghouse will "...establish standards for research and then 
determine. ..which studies meet those standards" (Taub, 2002). 

Like others in the educational community, counselors and counseling researchers 
welcome the opportunity to empirically test the validity of their practices and then, just 
as important, demonstrate that what they do makes a positive difference in students' 
lives. But they also have many concerns about the practical feasibility of implementing 
scientifically based studies. The purpose of this Digest is to provide an overview of the 
characteristics of scientifically based research, compare it to other types of research, 
and address some of the challenges and issues that are particularly relevant to 
conducting scientifically based studies of counseling interventions. To accomplish this, 
the authors have included selected portions of the ERIC/CASS publication. Research in 
Counseling & Therapy (Loesch & Vacc, 1997). 

WHAT IS SCIENTinCALLY BASED RESEARCH? 



In a New York Times article (November 1 0, 2002), James Taub pointed out that 
"Journalists using the most exacting method available to social science - that is, 
counting - have determined that the phrase 'scientifically based research' occurs more 
than 100 times in the No Child Left Behind Act of 2001 ." Other terms that are being 
used more frequently now by educators and policy makers include empirical evidence, 
empirical research, evidence based education and randomized control trials (or RCTs). 
Whatever term is used, the intent is to refer to experimental research studies that 
employ two fundamental procedures - the use of control groups and the random 
assignment of subjects to the different treatment groups. Typically, experimental 
research is undertaken when the research question involves a question of causality, i.e. 
whether changes in one variable cause changes in another variable (Hadley & Mitchell, 
1995). It is precisely this type of research, which will provide evidence of what works (or 
what works better). 
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CHARACTERISTICS OF EXPERIMENTAL 
RESEARCH METHODS 



Isaac and Michael (1981) identified the purpose of experimental research as 
investigating "possible cause-and-effect relationships by exposing one or more 
experimental groups to one or more treatment conditions and comparing the results to 
one or more control groups not receiving the treatment" (p.52). They also identified 
seven characteristics of experimental research implied in their definition: (a) 
management of the predictor and criterion variables along with the conditions in which 
the investigation is conducted; (b) use of a control group; (c) attempting to control 
variance among the predictor and criterion variables: (d) internal validity; (e) external 
validity; (f) ability to manage multiple predictor, criterion, and extraneous variables; and 
(g) exercise of control which makes experimental research powerful (but also somewhat 
artificial) when applied to human subjects. 

PREDICTOR AND CRITERION VARIABLES 



There are two primary types of variables in experimental research (Hadley & Mitchell, 
1995). The independent (aka treatment) variable is manipulated, managed, or 
administered by the researcher. The result of the manipulation is the measured or 
observed change in the dependent variable. While the terms independent and 
dependent variables have been used traditionally, the terms predictor and criterion 
variables are better descriptors, particularly in the context of experimental research. 
Predictor variables must be carefully chosen or designed to maximize differences due to 
their effects. Reliable and valid criterion variables must be selected or designed to 
accurately measure change caused by the predictor variables (Pickering, 1997). 

CONTROLLING VARIANCE 



Kerlinger (1973) offered an often-cited mnemonic to define what is meant by variance 
control in experimental research. MAXMINCON refers to MAXimizing the variance 
associated with the relationship between the predictor and criterion variables, 
MINimizing the error variance associated with measurement of the criterion variables, 
and CONtrolling extraneous variance attributable to other variables not included in the 
investigation. According to Kerlinger, maximizing the variance related to the interaction 
of the predictor and criterion variables requires designing the levels of the predictor 
variables to be as different from each other as possible. Minimizing the error variance is 
accomplished by controlling the conditions in which the investigation is conducted and 
choosing reliable measures of the criterion variables. Controlling extraneous variance 
may involve any of a variety of procedures, such as selecting a group of subjects who 
are homogeneous on the variable, randomly selecting and assigning subjects to groups, 
or perhaps adding the variable to the investigation as another predictor variable. 

INTERNAL VALIDITY 
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Internal validity refers to the level of confidence that the predictor variable(s), rather than 
an extraneous variable, produced the change found in the criterion variable(s). Hadley 
and Mitchell (1995), Heppner, Kivlighan, and Wampold (1992), and Isaac and Michael 
(1981) listed a variety of threats to internal validity including group composition, 
experimental mortality, history, maturation, practice effects, placebo effects, the 
Hawthorne effect, the John Henry effect, experimenter bias, demand characteristics, 
rater and observer effects, instrumentation, and statistical regression. For example, 
when subjects are selected because of group membership rather than being randomly 
selected, when subjects know they are part of an experiment and merely respond to 
receiving "special" treatment, or when the measuring instruments are not reliable and 
valid, it is doubtful whether the treatment variable caused the change in the criterion 
variable or whether it was caused by some other extraneous factor(s) (Pickering, 1997). 

EXTERNAL VALIDITY 



External validity is the degree to which the results can be generalized to other 
populations. Hadley and Mitchell (1995), Heppner, et al. (1992), and Isaac and Michael 
(1981) described a variety of threats to external validity including initial 
population-sample differences, mortality, artificial research arrangements, pretest 
influence, and multiple-treatment influences. For example, generalization to other 
populations probably will be limited if the sample chosen is not actually representative of 
the intended population, if subjects who leave the investigation differ in some way from 
those who remain, if subjects studied in laboratory settings perform differently than they 
do in naturally occurring situations, if pretesting sensitizes subjects to the treatment, or if 
multiple treatments are administered to each subject. Managing threats to external 
validity involves attempting to insure that both the subjects and the context in which the 
investigation is conducted are appropriately representative. 

RIGOR VERSUS RELEVANCE 



Experimental research almost always results in procedural compromise because control 
of one type of variance may cause problems in attempting to control another type of 
variance. Gelso (1979) labeled this "the bubble hypothesis," referring to the difficulty 
which arises when someone attempts to place a decal on a window and a bubble 
appears. When the bubble is depressed in one area, it arises somewhere else. Gelso 
also discussed how experimental rigor is related to internal validity. Threats to internal 
validity are most easily managed in controlled laboratory conditions, but human 
behavior rarely occurs in tightly controlled laboratory situations and thus generalization 
is limited. 



ISSUES IN DESIGNING AND IMPLEMENTING 
RESEARCH ON COUNSELING 



Experimental methods are often touted as the sine qua non of research in counseling. 
However, they are not always the most appropriate to answer questions. Heppner, et al. 
(1992) stated that just as there is no uniform method of counseling, there is no uniform 
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method for conducting research. They offered five guidelines to determine whether 
experimental methods should be used: (a) does the professional literature support the 
use of experimental methods or is a less rigorous approach more appropriate? (b) does 
the literature include a variety of research methods or have one or two methods 
dominated? (c) are sufficient resources available to support the type of research method 
desired? (d) can rigor and relevance be balanced to answer the research question? and 
(e) can responses to the previous four guidelines be balanced with each other? 
Whenever experimental methods are preferred or recommended to answer the 
research question, the issues presented should be considered carefully in designing the 
investigation. 

CONCLUSION 

In reviewing the characteristics of experimental research presented above, it is apparent 
that some aspects of counseling are not readily amenable to the application of scientific 
research methods. This is not to say that randomized control trials cannot or should not 
be undertaken when feasible. However, due to the policies of some organizations that 
preclude use of no treatment control groups and/or parental groups opposed to the 
involvement of their children in research, these studies will be limited. Given the many 
potential threats to validity, consumers of the results will need to carefully evaluate the 
quality of the evidence as well as its relevance to other counseling situations. Of 
particular interest to counselors is the comparison of the effectiveness of different 
counseling approaches in obtaining desired outcomes. Thus, an alternative to the 
treatment-no treatment design is a Treatment A vs. Treatment B design. However, even 
this design can produce flawed results due to differences in the implementation of 
treatments by individual counselors. Thus, in assessing the usefulness of results 
obtained from experimental studies, one must always consider which is better - to base 
actions on poor experimental research (i.e., research which has not fully met the rigors 
of controlling all of the relevant variables) or to base actions on good qualitative 
research (i.e., research that applies intensive and extensive analysis from a number of 
perspectives and sources). 

At the present time, it appears that those who teach and practice counseling will have to 
rely heavily upon a careful evaluation of the results of all types of research, well 
conducted meta-analyses of counseling research (e.g., Sexton, et.al., 1987), and best 
human judgments. In fact, in a speech delivered at a seminar where leading experts in 
the fields of education and science discussed the meaning of scientifically based 
research hosted by the Assistant Secretary for Elementary and Secondary Education, 
Feuer (2002) stated that, "...what scientists themselves often acknowledge is that there 
is a dimension of human judgment that can be missed with an overzealous focus on the 
rigor of scientific method." 
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